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ABSTRACT 

One of the important underlying assumptions of all 
item response theory (IRT) models is that of local independence. This 
assumption requires that the response to an item on a test not be 
influenced by the response to any other items. This assumption is 
often taken for granted, with little or no scrutiny of the response 
process required to answer individual test items. Ackerman and Spray 
(1986) proposed a dependency model with which the interaction of such 
factors as the amount and direction of item dependency, item 
difficulty and discrimination, and item order or sequence effects 
could be simulated. In the present study, item response data were 
generated with varying degrees of response dependency using their 
model. These data were used to determine the robustness of the IRT 
calibration programs LOGIST and BILOG to violations of local 
independence. Results suggest that calibrated dependent item 
parameters tend to be overestimates of the original item parameters. 
Ability estimates, however, were more affected as the degree of 
dependency increased. A description of the ACT Mathematics Usage Test 
is appended. (Author/JAZ) 
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Abstract 

One of the important underlying assumptions of ail item response theory 
models is that of Local independence. This assumption requires that tha 
response to an item on a test not be influenced by the response to any other 
items. This assumption is often taken for granted, with little or no scrutiny 
of the response process required to ansver individual test items. 

Ackerman and Spray (1986) proposed a dependency model vith which the 
interaction of such factors as the amount and direction of item dependency, 
item difficulty and discrimination, and item order or sequence effects jouid 
be simulated. In this study item response data were generated vith varying 
degrees of response dependency using their model. These data were used to 
determine the robustness of the IRT calibration programs LOGIST and BILOG to 
violations of local independence. Results suggest that calibrated dependent 
items parameters tend to be overestimates of the original item parameters. 
Ability estimates, however, were more affected as the degree of dependency 
increased* 
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The Robustness of LOGIST and BILOG I&T 
Estimation Programs to Violations of Local Independence 

Al though it is one of the basic as sump t ions underlying item response 
theory (iRT), local independence of item responses is often taken for granted, 
with little attention paid to determine if the process of responding to one 
item influences tha response(s) to other itemCs). Yet violations of this 
assumption can easily occur when several items are embedded in the same 
passage, or when items contain multiple parts. Local independence is violated 
whenever the response process of one item provides the necessary cognitive 
schema to trigger a response to a subsequent item* 

The purpose of this paper is to examine the robustness of the IftT 
calibration programs BILOG <Hislevy & Bock, 1984) and LOGIST (Wingersky, 
Barton, & Lord, 1982) to varying degrees of locai dependence. Dependency was 
imposed upon a set of real test data using the model developed by Ackerman and 
Spray (1986). 

Model Definition 

Ackerman and Spray C1986) proposed an item dependency model which is 
based upon a finite, two state £D or 1, incorrect or correct) Markov 
process. Xn the model P^(B^) is defined as the probability an examinee with 
ability, will answer item j correctly, independently of any other test 

item. ?.($,) can be determined using any response function, however For the 
purposes of this study, it was defined by the two parameter logistic 1ST 
model • 
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The dependency model is defined by a transition matrix between any two 



items, j - 1 and j, in a k~item teat: 



jth item 



Q 



1 



0 



1 - a P - ( 9 , ) 



J i 



jth - 1 item 



1 




1 - 0Q.(9< ) 



In this model, aP^CsO represents the probability that an examinee with 
trait 6^ will "move" from ad incorrect response on item j - 1 (state 0) to a 
correct response on item j (state l). Likewise, SQ^Ce^) represents a 
transition probability from a correct response on item j - 1 no an incorrect 
response on item j. The probabilities 1 - aP^(Q^) and 1 - 0Qj(eO imply state 
consistency between item responses* It should be noted that items j and j - 1 
need not be adjacent items, but may occur anywhere throughout the test. 

The parameters a and S are dependency weights where 0 < a < 1 and 0 < S < 1. 
These dependency weights may take on any value within the specified range and need 
not be equal. The weights can be assigned values independently of one another, and 
thus fix the direction of state transition (e.g. increasing or decreasing the 
probability of going from a correct to an incorrect response). When a = & = 1, the 
items are totally independent, and when a. = E = 0, the items are completely 
dependent * 

The probability of answering the jth it^m correctly, given an incorrect 



or a correct response to the previous item is defined as P-{B.) where: 
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p 1 . <e.) - q! , (e,)ap ( (0, + p! . ce-) {i - sq.<9.)} 

J x J - 1 1 J I j - 1 1 1 % jl J 

and 

t 

P. _ . is the probability of a correct response on item j - 1 and 



i i 




The degree to which local independence is violated depends upon several 
factors including the a and (3 weight s , the indi vidua 1 i tem parameters C i . e . , 
difficulty and discriminat ion) of those items with in the dependent sequence ? 
the order of the items Ce*g- easy to difficult, difficult to easy), and the 
length of the dependent sequence. Spray and Ackerman (19S6) summarized all of 
these factors in terms of a statistic For a dependent sequence of length 
m f $ is the sum of the absolute differences between the likelihood of each of 
the possible 2 m response patterns which can occur under a joint density 
function of the Markov process and the likelihood under an assumption of local 
independence for the m items. 

The absolute value of the differences is evaluated at some 
value B = 0£ Cwhich is thought to be representative of the entire examinee 
population) and summed over all possible 2 m response patterns. 

Specifically, 

where 

p *<H a ■ «J B o> = pCu i = u il e o )p < u 2 - " a lv u i> ••• 




and 
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It can be shown that for any given ability value, 0 O , 0.0 * 4> < 2,0, regardless 
of whether Pj{& 0 ) is defined as a one, two, or three parameter IRT model* 
When tf> = 0,0, local independence holds throughout the m item sequence, 
As 4> increases , the degree of dependency inc ceases » 

Alcr r T gh the 4> statistic is thought to be useful in describing the degree of 
dependency within a set of items, it is doubtful that it can be recovered in an 
estimation sense. Than is, $ is based on all possible response patterns for a 
"representative" value of 9, while in reality each value of 0 may be unique and 
be represented by only one response pattern ^rhich may or may not be the most 
likely. Thus, is believed to be more of a tool for describing dependent: data 
rather than an estimable parameter* 

In summary, this study had two main objectives. The first objective was to 
valid the use o£ the dependency model developed by Ackerman and Spray (1986) as a 
useful tool for generating dependent data. The second goal was to determine the 
affect various degrees of dependency had on the IRT calibration process using 
different sample sizes ■ 

Method 

To accomplish these two objectives, response vectors to a calibrated set of 
item parameters were generated using four levels of dependency with three 
different sample sizes, producing 12 different data sets. Sets were first 
examined to insure thai four levels of dependency existed in the created response 
sets* Then each of the 12 sets were calibrated using a two parameter logistic 
model with the computer programs BILOG and LOGIST* Ability and item estimates 
were examined to determine how dependency effected the estimation process. 
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Data Generation 



In this study data were generated using four Levels of dependence (no 
dependence, weak, medium, and strong) with three different Sample siaes {N = 400, 
800, and 1200)* The degrees of dependency were selected to be representative of 
nearly the full range of dependency as described by the $ statistic. The 
specific sample sizes were chosen for three reasons; it was thought that such 
sample sizes were realistic in terms of academic testing situations; it was felt 
that the sample size of 400 would be a minimum for use with the two parameter IRT 
model; and, that differences in calibration accuracy would be most evident in 
sample sizes having this range. 

Data were generated using the ACT Assessment Math Usage Form 26A aa a 
model. It is a 40 item multiple choice test with each item having five 
alternatives. A brief description of the content of the test is provided in 
Appendix A, 

The dependence sequence length of eight items was chosen because it was 
thought to be typical of the number of items assigned to a test passage. Items 
1"8 were selected to be the dependent block of items. The dependency weights 
and $ values for the four levels of dependency were: 



Level of Dependency 

$ a B 

Total independence 0.00 1,00 1-00 

Weak 0,66 0.60 0,80 

Medium 1,09 0,50 0-40 

Strong 1,57 0.30 0.10 
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Using previous calibrated LOGIST two parameter logistic item parameter 
estimates* and randomly generated abilities from a N(Q, I) distribution, 1,000 
response vectors for the 12 datasets were generated using the dependent model. 
Each dataset was then calibrated separately using the IRT calibration programs 
LOGIST and BILOG* The two IRT calibration programs use different estimation 
procedures* LOGIST uses a joint maximum likelihood estimation procedures, 
whereas BILOG uses marginal maximum likelihood estimation* The default method of 
scoring subjects was selected for all BILOG computer runs. This method of 
scoring was expectation a posteriori using a normal N(0, 1) Bayesian prior. The 
default log-normal prior was were also used in the item discrimination 
calibration. No prior was used in estimating the difficulty parameters. 

Mean inter-item tetrachoric correlations for the dependent and independent 
items are shown in Table 1. In the total independent datasets the mean 
correlations ranged from .383 to ,433 suggesting a moderate degree of similarity 
among response patterns for the original 40 items. The effect of the dependency 
model is clearly demonstrated. As compared to the independent case, the average 
tetrachoric increases as the level o£ dependency increases. 



Insert Table I about here 



After each calibration run, Yen's Q3 
used as a compari son measure to determine 
were to violations of Local independence, 
over examinees of: 



statistic (Yen, 198A) was computed and 
how robust the calibration programs 
Q-j is defined as the correlation taken 
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d- = u. - P. (a ) and 
1 ■ 1 ■ i 1 



d. = u. - p. (e ) 
j- J- j 



where u. is the score of an item on item i» 

l ■ 



u^ is the score of an item on item j 

and P. (9 ) f P, (9 ) are the probabilities of a correct 

1 J 

response based upon the estimated item and ability 
parameters, A Fisher r-to-z transformation of Q3 is 
approximately distributed as a normal variable with mean 
equal to zero and variance equal to l/(N-1), 

Average absolute differences CAAD) between the calibrated items parameters 
for the independent datasets and the dependent datasets ^ere computed as a 
measure of item parameter shift* 

To estimate bias in the estimated abilities, bias (ft - 8} was plotted for 
each dstaset within each calibration program. Correlations between the ability 
estimates for each level of dependency with equal sample sizes were also 
calculated • 



Results 



The average p value and average biserial correlation for items 1-8 and items 
9-40 for each level of dependency are shown in Table 2. As the degree of 
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dependency increased the average difficulty (p) increased. This is due in part 
to ths order of difficulty within the dependent block of items, That is, if an 
easier item precedes and is dependent with a subsequent item, it will have a 
tendency to make Che subsequent item easier, (see Ackerman L Spray, 1986, p. 12- 
13), Thus, subsequent items tend Co become more similar in difficulty to the 
previous items on which they are dependent* It can also be seen in Table 2 that 
the items within the dependent block become more homogeneous as the dependency 
increases. (Note biserial correlations were computed using the item score with 
the total test score.) 



Insert Table 2 about here 

The AAD values and correlations between the estimated item parameters for 
LOGIST and BILOG are displayed in Tables 3a and b. The AAD values increased 
dramatically for each estimation computer program as the dependency increased. 
This was more true of BILOG than LOGIST. Difficulty differences, though 
affected* appear to increase at about half the rate of the discrimination 
differences. There was a slight increase in the LOGIST estimated discrimination 
differences for the independent items (compared to the * = 0 case). This is 
probably due to dependency contamination from the iterative ability/item 
e s t ima t i on pr oc e s s . 



Insert Tables 3a and b about here 
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Yen T s Q3 analysis of the calibrated parameters are shown in Table 4. The 
pattern observed by Yen (1984) can also be observed in these data. That is, the 
more Locally dependent the items the higher the Q3 value. By comparing the Q3 
computed on the calibrated dependent items data with the Q3 based on the 
independent item sets one can obtain a relative sense of how rubust each 
calibration program is to the violation of Local independence. The Q3 value for 
the dependent item block increased for each sample size as the dependency Level 
increased. For each Level of dependency the <J3 value for the H = 800 case was 
Larger thar the two other sample siaes, except for the BILOG estimation of the 
strong dependency case. This might be a result of sampling variance. However, 
the Overall Q3 values computed using BILOG estimates do not increase as much as 
though computed using the LOGIST estimated values. 

It should be noted that us. : n£ Bock's x 2 goodness-of-f i t measure (Bock & 
MisLevy, 1982), each calibration pvogrim's item p, -ameter estimates fit the ?PL 
IRT model for all datasets at the .01 Level of significance* 



Insert Table 4 about here 



Correlations between estimated abilities for the no dependency case and the 
three other levels of dependency for each Calibration program £or each sample 
size showed little difference* For each sample size as the dependency increased, 
the correlation between B and G dropped from .95 to ,90- This suggests that if 
the violation of Local independence is great enough, ability estimation can be 
affected. Ho difference vas detected for the ability estimates between the tvo 
calibration programs. 
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Bias pLots of 6 - 9 for each of L0G1ST and B1L0G calibration runs are shown 
ia Figures la, bi and c. The fl's were rank ordered and divided into quantiles in 
.2 increments from "3.0 to +3.0. All estimated abilities were repealed to 
the 0 scale. Differences between B - 6 were obtained for each group and then 
plotted. As dependency increased both estimation programs appeared to 
increasingly overestimate abilities at the lower end of the ability scale and 
underestimate abilities at the upper end of the scale are to be expected because 
the dependency was injected into the easier iteps on the test causing them to 
become even easier. Less variance in the 0 - e difference was detected for the 
sample size of 800, then for N = 400, or N = 1200. 



Insert Figures la* b, and c about here 



Cone lus ions 

The results of this study have several implications for the calibration of 
locally dependent items. First, the stronger the violation of local independence 
(as defined in the model by Ackerman and Spray, 1986) the greater the effect of 
item parameter calibration regardless of sample siae. If items are calibrated in 
a dependent sequence and used separately (e.g., in an adaptive testing pool), the 
overestimates/underestimates of the parameters could affect ability estimation. 

In this study, dependency was injected into an eight item sequence at the 
beginning of the test where the items were the easiest. Thus, ability estimation 
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was affected at the Vow end of the ability scale. If, however, this dependency 
had been imposed on the middle difficulty or very hard items, it is believed that 
ability estimates at the middle and upper ends of the scale would be affected. 

Very little difference was noted between BILOG and LOGIST' s ability 
estimation* Within each Level of sample size, the 9 LOGI5T and & BILOG tended to 
correlate in the neighborhood of .98 to .99.. Bias plots of the estimated 
abilities revealed that both programs are affected as the dependency increases, 
regardless of sample size. BILOG resolves violations of local independence by 
overestimating the discrimination parameters more so than LOGIST* 

Yen's Q3 statistic validated the Ackerman and Spray dependency model as a 
useful tool to use for simulating dependent data. The Q3 results suggest that 
BILOG is Slightly better than LOGIST at calibrating the response data to a two 
"parameter cognitive model assuming local independence. Perhaps by imposing 
appropriate Bayesian priors on the ability and item distribution the BILOG 
calibration process could be improved. 

These results strongly suggest that the calibration of item parameters 
should be conducted jointly with a review of the response processes for all items 
on the calibrated test. If after studying the response ptrocess required for each 
item violations of local independence are suspected, calibration results should 
be guarded. 

This study needs to be replicated to verify the findings. Other directions 
for future research would be to impose dependency with items of differing 
difficulty levels to see if there is an effect on ability estimation at other 
points of the ability scale* Likewise, the results should be replicated with a 
3PL mod** to study the effect of guessing. 
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Table 1 

Mean interitero tetcac l: 0Ti j» *^v"cc lat lQi'.q tor items in the dependent and independent 
item blocks 



* 






r 

tetrd 






Items 1-8 
(Dependent Block) 


Items 9-32 
(independent Block) 


0.00 


400 


.433 


.488 


xno dependency^ 


800 


.383 


.477 




1200 


.429 


.476 


0.66 


400 


.553 


.488 


(weak) 


800 


.481 


.477 




1200 


.527 


.476 


1.09 


400 


.644 


.488 


(medium) 


800 


.594 


,477 




1200 


.645 


.476 


1-57 


400 


.827 


.488 


(strong) 


800 


.855 


.477 




1200 


.866 


.476 



ERIC 
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Table 2 



Mean difficulty, biaerial correlations^ and reliability coefficients for the 



dependent (p) and independent (i) items tor each level of dependency 







P 




r bi3 






D 




I 


D 


r 


KR-20 


0.0 


.61 




.36 


.66 


.73 


.94 


0.6 


.64 




.36 


.68 


.73 


.94 


1.1 


.69 




.36 


.71 


.73 


.94 


1.5 


.79 




.36 


.79 


.72 


.94 



Note. Sample size was 1200. 
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Tabic 3a 



Correlations and average absolute differences of the dependent (p) and independent 



(i) items for values of LOCIST calibrated a, b and a, b for different dependency 



conditions 



DIDI DIDI 



0.0 


400 


.82 


.94 


.98 


.97 


.16 


.11 


.07 


.08 




800 


.91 


.98 


.98 


.98 


.10 


.10 


.06 


.05 




1200 


.87 


.98 


.99 


.99 


.13 


,07 


.03 


.04 


0.6 


400 


.87 


.93 


.90 


.97 


.14 


,12 


.13 


.10 




800 


.94 


.97 


.93 


.98 


.07 


,07 


.13 


.07 




1200 


.85 


.93 


.93 


.99 


.11 


.07 


.12 


.05 


1.1 


400 


.60 


.94 


.82 


.97 


.37 


.12 


.23 


.08 




800 


.50 


.96 


.76 


.98 


.23 


.13 


.25 


.06 




1200 


.44 


.98 


.90 


.99 


.39 


.08 


.19 


.04 


1.5 


400 


-.08 


.94 


-.17 


.97 


.95 


.17 


.33 


.08 




800 


.02 


.98 


-.21 


.98 


1.04 


.19 


.34 


.08 




1200 


-.24 


.98 


-.09 


.99 


1.25 


.13 


.31 


.05 
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Table 3b 

Correlations and average absolute differences of the dependent <D) and independent 
(I) items between values of BILO n calibrated at b and a, b for different dependency 
conditions 



* 


N 


r 

a a 




r bb 








yib-b| 
k 






D 


I 


D 


I 


D 


I 


D 


I 


0.0 


400 


.82 


.95 


.99 


.98 


.76 


.71 


.06 


.08 




800 


.92 


.97 


.98 


.99 


.60 


.72 


.05 


.05 




1200 


.87 


.99 


.99 


.99 


.74 


.70 


.03 


.04 


0.6 


400 


.86 


.94 


.90 


.98 


.74 


.58 


.13 


.09 




800 


.92 


.98 


.93 


.99 


.48 


.58 


.13 


.06 




1200 


.87 


.98 


.93 


.99 


.66 


.57 


.12 


.06 


1.1 


400 


.51 


.94 


.83 


.98 


1.16 


.72 


.23 


.OS 




800 


.48 


.98 


.76 


.99 


.80 


.83 


.26 


.06 




1200 


.43 


.98 


.91 


.99 


1.19 


.71 


.20 


.04 


1.5 


400 


-.17 


.95 


-.21 


.98 


2.38 


.82 


.33 


.07 




800 


-.09 


.98 


-.23 


.98 


2.87 


.77 


.33 


.07 




1200 


-.15 


.98 


.17 


.99 


3.66 


.74 


.30 


.04 
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Table 4 

Yen'g Q3 analysis of dependent and independent item blocks calibrated with LOGIST 
and BTLOG 



SI 









LOGIST 




BILOG 




M 


DEP 


IfID 


DEP 


IfID 


0.0 


400 


- .042 


-.024 


-.032 


-.012 




800 


-.025 


-.024 


-.013 


-.012 




1200 


-.039 


-.024 


-.018 


-.013 


0.6 


400 


.040 


-.02 7 


.053 


-.015 




800 


.053 


-.027 


.076 


-.014 




1200 


.040 


-.027 


.061 


-.014 


1.1 


400 


.100 


-.029 


.106 


-.019 




300 


.137 


-.030 


.160 


-.021 




1200 


.113 


-.029 


.113 


-.020 




400 


.267 


-.034 


.206 


-.023 




800 


.303 


-.036 


.192 


-.021 


1.5 


1200 


.251 


-.033 


.124 


-.017 
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Figure Captions 

Figure La . Bias plot of 9 - 6 for LOG I ST and BILOG calibrated abilities with 
different levels of item dependency; N = 400. 

Figure Lb . Bias plot of 9 - B for LOGIST and BILOG calibrated abilities with 
different levels of item dependency; N = 800. 

Figure Lc . Bias plot of 9 - e for LOGIST and BILOG calibrated abilities with 
different levels of item dependency; N = 1200. 
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Bias Plot of Logist 
Ability Estimates N = 800 
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Ability Estimates N = 800 
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Bias Plot of Logist 
Ability Estimates N = 1200 




Appendix A 



ACT MATHEMATICS USAGE TEST 

Description or the lest The Mathematics Usage Test fe a 40-ilem. 50-minuie test thai measures ihe students 1 
mathematical reasoning ability It emphasizes the solution or practical quantitative problems thai arc 
encountered in many poslsecondary cuTricula and includes a sampling of mathematical techniques covered in 
high school courses. The lest emphasizes quantitative reasoning, rather than memorization of formulas, 
knowledge of techniques, or computational skill. Each iiem in ihe test poses a question with five alternative 
answers* the last of which may be "None of ihe above.* 

Content of Ihe tesl* In general, the mathematical skills required for the icst involve proficiencies emphasised in 
high school plane geometry and first- and second-year algebra Six types of LOnicnt are included in ihe test. 
These categories and ihe approximate proportion of ihe tesi devoted lo each are given belov;. 



Mathematics Content Area Proportion of Test Number of Items 

a. Arithmetic and Algebraic Operations ,10 4 

fe. Arithmetic and Algebraic Reasoning .35 14 

c. Geometry .20 8 

d. Intermediate Algebra .20 8 
c. Number and Numeration Concepts . 10 4 
f. Advanced Topics .05 2 

Total 1 00 40 



a. Arithmetic and Algebraic Operations, The items in this category explicitly describe operations LQ be 
performed by the student. The operations include manipulating and simplifying expressions containing 
arithmetic or algebraic fractions* performing basic operations in polynomials* solving linear equations in One 
unknown, and performing operations on signed numbers. 

b. Arithmetic and Algebraic Reasoning, These word problems present practical situations in which algebraic 
and/or arithmetic reasoning is required. The problems require the student to interpret the question and either 
to solve the problem or to find an approach to its solution. 

c. Geometry. The items in this category cover such topics as measurement of lines and plane Surfaces, properties 
of polygons, the Pythagorean theorem, and relationships involving circles. Both formal and applied problem*; 
ate included. 

d. Intermediate Algebra The items in this category cover such topics as dependence and variation of quantities 
related by specific formulas, arithmetic and geometric series, simultaneous equations, inequalities, exponents, 
radicals, graphs of equations, and quadratic equations. 

e. Number and Sumtration Concepts. The items in this category cover such topics as rational and irrational 
numbers, set properties and operations, scientific notation, prime and composite numbers, numeration 
systems with bases oilier than 10. and absolute value. 

f. Advanced Topics. The items in this category cover such topics as trigonometric functions, permutations and 
combinations, pTobabfiiH. siaiittics. and logic. Onl> simple application* of the skills implied b> these topics 
are tesied. 
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